Evaluation of neural models’ linguistic competence: evidence from Russian predicate agreement

This study investigates the linguistic competence of modern language models. Artificial neural networks demonstrate high quality in many natural processing tasks. However, their implicit grammar knowledge remains unstudied. The ability to judge a sentence as grammatical or ungrammatical is regarded key property human’s competence. We suppose that models’ also occurs grammaticality sentence. In order test networks’ competence, we probe acquisition number predicate agreement Russian. A dataset consisted artificially generated and sentences was created train Automatic generation allows us particular phenomenon, detach from vocabulary pragmatic differences. use transfer learning pre-trained networks. results show all considered models accuracy Matthew's correlation coefficient values which can be attributed successful rules. classification reduced for with inanimate nouns nominative-accusative case syncretism. complexity syntactic structure turns out significant Russian model Slavic languages, but it does not affect errors distribution multilingual